Method and device for obtaining 3d images

ABSTRACT

A method and device are provided for obtaining a 3D image. The method includes sequentially projecting a plurality of beams to an object, each of the plurality of projected beams corresponding to a respective one of a plurality of sectors included in a pattern; detecting a plurality of beams reflected off of the object corresponding to the plurality of projected beams; identifying time-of-flight (ToF) of each of the plurality of projected beams based on the plurality of detected beams; identifying a distortion of the pattern, which is caused by the object, based on the plurality of detected beams; and generating a depth map for the object based on the distortion of the pattern and the ToF of each of the plurality of projected beams, wherein the plurality of detected beams are commonly used to identify the ToF and the distortion of the pattern.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2018-0103662, filed on Aug. 31,2018, in the Korean Intellectual Property Office, and Korean PatentApplication No. 10-2018-0112975, filed on Sep. 20, 2018, in the KoreanIntellectual Property Office, the disclosure of each of which isincorporated by reference herein in its entirety.

BACKGROUND 1. Field

The present disclosure relates generally to methods and devices forobtaining three-dimensional (3D) images, and more specifically, tomethods and devices for obtaining the depths of objects or scenes.

2. Description of Related Art

The growth of augmented reality (AR) and virtual reality (VR) technologyled to an increasing interest in three-dimensional (3D) cameras. 3Dcameras may be installed in various electronic devices, such as mobiledevices or computers, or may be provided as standalone electronicdevices.

3D cameras may adopt a stereoscopic passive or active projective schemein order to obtain 3D images. Stereoscopic passive schemes may generatea sparse or semi-sparse depth map for an environment. However,stereoscopic passive schemes are inefficient under low-texture andlow-illumination conditions. Creating a dense depth map requires highercalculation complexity.

To generate a depth map, active projective schemes may use atime-of-flight (ToF) of light projected from the 3D camera or adistortion of a pattern projected from the 3D camera. In using the ToF,the depth map may be obtained based on distances between multiple pointsand a scene calculated based on a plurality of ToFs. However, thisscheme may exhibit significant depth errors within a short range.

A pattern distortion scheme may obtain a depth map by projecting aparticular pattern onto an object or scene, detecting the patterndistorted by the object or scene, and then basing the depth map on thedetected pattern distortion. Triangulation may be used to calculate adistance between the pattern distortion and the object. The patterndistortion scheme may also be referred to as a triangulation techniqueor a structured light (SL) scheme, as the pattern is regarded as SL.Errors in the triangulation technique may be minor within a short rangebut often become serious as the range expands due to limitations oftriangulation itself.

As such, it is often difficult to apply conventional techniques forobtaining 3D images to AR/VR applications that require a higher-qualityof depth map in a wide depth range.

SUMMARY

Accordingly, an aspect of the present disclosure is to provide a methodand a device for adaptively obtaining a higher-quality depth map in awider depth range.

In accordance with an aspect of the present disclosure, a method isprovided for obtaining a 3D image by a device including a beam projectorand a beam detector. The method includes sequentially projecting, fromthe beam projector, a plurality of beams to an object, each of theplurality of projected beams corresponding to a respective one of aplurality of sectors included in a pattern; detecting, by the beamdetector, a plurality of beams reflected off of the object correspondingto the plurality of projected beams; identifying ToF of each of theplurality of projected beams based on the plurality of detected beams;identifying a distortion of the pattern, which is caused by the object,based on the plurality of detected beams; and generating a depth map forthe object based on the distortion of the pattern and the ToF of each ofthe plurality of projected beams, wherein the plurality of detectedbeams are commonly used to identify the ToF and the distortion of thepattern.

In accordance with an aspect of the present disclosure, a device isprovided for obtaining a 3D image. The device includes a beam projectorconfigured to sequentially project a plurality of beams to an object,each of the plurality of projected beams corresponding to a respectiveone of a plurality of sectors included in a pattern; a beam detectorconfigured to detect a plurality of beams reflected off of or the objectcorresponding to the plurality of projected beams; and a controllerconfigured to identify ToF of each of the plurality of projected beamsbased on the plurality of detected beams, identify a distortion of thepattern, which is caused by the object, based on the plurality ofdetected beams, and generate a depth map for the object based on thedistortion of the pattern and the ToF of each of the plurality ofprojected beams, wherein the plurality of detected beams are commonlyused to identify the ToF and the distortion of the pattern.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing detailed description taken in conjunction with theaccompanying drawings, in which:

FIG. 1 is a block diagram illustrating a device for obtaining a 3D imageaccording to an embodiment;

FIG. 2 is a block diagram illustrating a beam projector according to anembodiment;

FIG. 3 is a flowchart illustrating a method for obtaining a 3D imageaccording to an embodiment;

FIG. 4 illustrates a pattern for projecting a plurality of beamsaccording to an embodiment;

FIG. 5 illustrates a pattern for detecting distortion according to anembodiment;

FIG. 6 illustrates a pattern for detecting ToF according to anembodiment;

FIG. 7 illustrates an example of modifying resolution of a patternaccording to an embodiment;

FIG. 8 illustrates an example of pattern shifting according to anembodiment;

FIG. 9 illustrates a pattern according to an embodiment;

FIG. 10 illustrates an order of projecting beams based on a patternaccording to an embodiment;

FIG. 11 illustrates an example of projecting a beam and detecting areflection according to an embodiment;

FIG. 12 illustrates a method for projecting a beam using a sync signalaccording to an embodiment;

FIG. 13 illustrates a method for detecting ToF according to anembodiment; and

FIG. 14 illustrates a method for obtaining the distance to an objectbased on SL according to an embodiment.

Throughout the drawings, like reference numerals may refer to likeparts, components, structures, etc.

DETAILED DESCRIPTION

Various embodiments of the present disclosure will now be described indetail with reference to the accompanying drawings. In the followingdescription, specific details such as detailed configuration andcomponents are merely provided to assist the overall understanding ofthese embodiments. Therefore, those skilled in the art will understandthat various changes and modifications of the embodiments describedherein can be made without departing from the scope and spirit of thepresent disclosure. In addition, descriptions of well-known functionsand constructions are omitted for clarity and conciseness.

Although numerical terms, such as “first” and “second” may be used todescribe various components, the components are not limited by theterms. These terms are provided simply to distinguish one component fromanother. Accordingly, a first component as described herein may also bereferred to as a second component within the technical spirit of thedisclosure, and vice versa.

FIG. 1 is a block diagram illustrating a device for obtaining a 3D imageaccording to an embodiment.

Referring to FIG. 1, a device 100 for obtaining 3D images may be a 3Dcamera or an electronic device including a 3D camera. The device 100includes a beam projector 110, a beam detector 120, a controller 130,and a memory 140.

The beam projector 110 may generate a beam and project the beam to ascene 150 or an object. The scene 150 may denote an area where beamsgenerated by the beam projector 110 are projected to generate a 3Dimage. An object may denote a thing for generating a 3D image. The beamgenerated by the beam projector 110 may be an infrared ray (IR) beam,but without being limited thereto, may also be a visible or ultraviolet(UV) beam or other various frequencies of beams.

FIG. 2 is a block diagram illustrating a beam projector according to anembodiment.

Referring to FIG. 2, the beam projector 110 includes a laser 111, alaser driver 114, a first mirror 112, a second mirror 113, and amicroelectromechanical system (MEMS) driver 115. The laser 111 maygenerate and emit laser beams of a preset frequency. The operation ofthe laser 111 may be controlled by the laser driver 114. The laser beamemitted from the laser 111 is directed to the first minor 112, whichreflects the laser beam to the second mirror 113. For example, the firstmirror 112 may be a fixed minor, but is not limited thereto. The secondmirror 113 reflects the laser beam from the first mirror 112 to thescene 150. The second mirror 113 may be rotated to control the directionof the beam B1 emitted from the beam projector 110. For example, thesecond mirror 113 may be rotated around two axes.

A MEMS may be used to operate the second mirror 113. The MEMS driver 115may control the rotation of the second mirror 113 using the MEMS.

Referring back to FIG. 1, the beam detector 120 may detect the beam B2,which is emitted from the beam projector 110 and reflected by the scene150. The beam detector 120 may detect the presence or absence of thereflected beam B2 and its brightness or intensity. For example, the beamdetector 120 may detect IR beams. However, the frequency band of beamsdetectable by the beam detector 120 is not limited to IR bands.

The beam detector 120 may be configured to detect the frequency band ofthe beam B1 emitted from the beam projector 110. The beam detector 120may detect the signal of the frequency band of the reflected beam B2.The beam detector 120 may be a two-dimensional (2D) camera capable ofcapturing a particular area (e.g., the area of the scene 150). The 2Dcamera may include an array of photosensitive pixels. For example, thephotosensitive pixels of the 2D camera may be divided into pixels fordetecting the ToF and pixels for detecting the reflected pattern.Alternatively, at least one of the photosensitive pixels of the 2Dcamera may be used to detect the ToF and the reflected pattern.

The controller 130 may control the operations of the beam projector 110and the beam detector 120. The controller 130 may calculate and generatea depth map for the scene 150 or object based on the reflected beam B2detected by the beam detector 120.

The memory 140 may store instructions to operate the controller 130. Thememory 140 may store permanent or temporary data necessary to operatethe controller 130. The memory 140 may store information about one ormore patterns used for the beam projector 110 to project the beam B1.

FIG. 3 is a flowchart illustrating a method for obtaining a 3D imageaccording to an embodiment. For example, the method of FIG. 3 may becarried out by a device, such as the device 100 as illustrated in FIG.1, for obtaining 3D images, or more specifically, the method of FIG. 3may also be performed by a controller, such as the controller 130 asillustrated in FIG. 1, which substantially controls the operations ofthe components of the device 100 for obtaining 3D images.

Referring to FIG. 3, in step 310, the device 100 sequentially projects aplurality of beams based on a pattern.

In step 320, the device 100 detects reflections of the plurality ofbeams, e.g., beams reflected off of the scene 150.

In step 330, the device 100 identifies respective ToFs of the pluralityof projected beams, based on the detected reflections of the pluralityof beams.

In step 340, the device 100 identifies a distortion of the pattern, andin step 350, the device 100 generates a depth map based on the ToFs andthe pattern distortion.

FIG. 4 illustrates a pattern for projecting a plurality of beamsaccording to an embodiment. For example, the pattern in FIG. 4 may beused to sequentially project a plurality of beams in step 310. Referringto FIG. 4, a pattern 400 is split into a plurality of sectors (orsections) 410. Each of the sectors 410 of the pattern 400 is denoted bya pixel. The beams may be projected corresponding to their respectivesectors. The beams corresponding to darker sectors of the plurality ofsectors 410 and the beams corresponding to brighter sectors of theplurality of sectors 410 may be modulated differently. For example, thebeams corresponding to the darker sectors and the beams corresponding tothe brighter sectors may differ in at least one of brightness,intensity, size, diameter, and/or frequency.

Spots 420 indicate sectors for measuring ToF among the plurality ofsectors 410. ToFs may be obtained based on the reflected beams of thebeams projected corresponding to the sectors for measuring ToF. AlthoughFIG. 4 illustrates that all of the plurality of sectors 410 are used formeasuring ToF, alternatively, only some of the plurality of sectors 410may be used for measuring ToF.

According to an embodiment, a pattern, such as the pattern 400, may begenerated by merging a first pattern for detecting a distortion ofindividual patterns and a second pattern for detecting ToF.

FIG. 5 illustrates a pattern for detecting distortion according to anembodiment, and FIG. 6 illustrates a pattern for detecting ToF accordingto an embodiment.

Referring to FIGS. 5 and 6, the device 100 (or the controller 130) maygenerate the pattern 400 by merging a first pattern 500 and a secondpattern 600, e.g., stored in the memory 140. Accordingly, the device 100may generate a pattern for projecting beams by selectively merging oneof a plurality of first patterns for detecting pattern distortion withone of a plurality of second patterns for detecting ToF, therebyadaptively and easily generating and applying various pattern shapes.

According to an embodiment, a pattern for projecting beams may bemodified and generated from an original pattern. For example, the device100 may generate the pattern for projecting beams from an originalpattern stored in the memory 140.

The modified pattern may be generated by modifying the resolution of theoriginal pattern.

FIG. 7 illustrates an example of modifying resolution of a patternaccording to an embodiment.

Referring to FIG. 7, a modified pattern 720 may be generated byincreasing the resolution of an original pattern 710. Although notillustrated in FIG. 7, each of the plurality of sectors split from themodified pattern may be smaller in size than each of the plurality ofsectors split from the original pattern 710. The modified pattern may begenerated by decreasing the resolution of the original pattern.

A modified pattern may be generated by shifting an original pattern.

FIG. 8 illustrates an example of pattern shifting according to anembodiment.

Referring to FIG. 8, a modified pattern 820 may be generated by shiftingan original pattern 810 in a right and downward direction. The directionof shifting may differ according to embodiments.

A plurality of modification methods may simultaneously be used togenerate the modified pattern.

The resolution of a pattern for projecting beams may be uneven in thepattern.

FIG. 9 illustrates a pattern according to an embodiment.

Referring to FIG. 9, a pattern 900 includes a first portion 910 with afirst resolution and a second portion 920 with a second resolution,which is lower than the first resolution. For example, the first portion910, which has a relatively larger resolution, may correspond to aregion of interest (ROI). The pattern 900 may be generated by mergingthe original pattern and the pattern obtained by modifying theresolution of the original pattern.

In step 310 of FIG. 3, the plurality of beams may sequentially beprojected corresponding to their respective sectors from the pluralityof sectors.

FIG. 10 illustrates an order of projecting beams based on a patternaccording to an embodiment.

Referring to FIG. 10, the plurality of beams are projected in a zigzagshape. Alternatively, the plurality of beams may be projected verticallyin a zigzag shape. Although projecting in a zigzag shape may beadvantageous in light of the efficiency of the operation of the beamprojector 120 to control the position of beams projected, the actualorder of projecting the plurality of beams may be varied as necessary.

FIG. 11 illustrates an example of projecting a beam and detecting areflection according to an embodiment.

Referring to FIG. 11, the laser 111 may generate a beam (a laser),corresponding to one of the plurality of sectors constituting a pattern.The beam emitted from the laser 111 is reflected by the first mirror 112and directed to the second mirror 113. The horizontal angle θ andvertical angle φ of the second mirror 113 may be adjusted. Thehorizontal angle θ and vertical angle φ of the second angle 113 may beset for the first beam B1 reflected by the second mirror 113 to bedirected to a desired location. The horizontal direction of the firstbeam B1 may be controlled by the horizontal angle θ, and the verticaldirection of the first beam B1 may be controlled by the vertical angleφ. The horizontal angle θ and vertical angle φ of the second mirror 113may be set for the first beam B1 to be directed to a location within thescene 150 corresponding to the sector corresponding to the beam amongthe plurality of sectors of the pattern.

The second beam B2, which is the reflection of the first beam B1 by thescene 150, may be detected by the beam detector 120, e.g., in step 320FIG. 3.

After projecting the beam corresponding to one of the plurality ofsectors of the pattern and detecting the reflected beam by theabove-described method, a beam corresponding to a next sector may beprojected such that its reflection may be detected, e.g., in the zigzagorder illustrated in FIG. 10.

By repeating the process, the projection of the beams corresponding tothe plurality of sectors of the pattern and the detection of theirreflections may be fulfilled.

When sequentially projecting a plurality of beams based on a pattern,such as illustrated in step 310 of FIG. 3, all, or at least some, of theplurality of beams projected may be jointly used to generate a depth mapusing the ToF scheme and the SL scheme. For example, when a firstpattern for detecting pattern distortion and a second pattern fordetecting ToF, which are merged to generate a pattern for sequentiallyprojecting a plurality of beams, differ in resolution or arrangement ofsectors, all of the beams projected do not need to be used to generate adepth map using the ToF scheme or a depth map using the SL scheme.Instead, at least some of the plurality of beams may be jointly used togenerate a depth map using the ToF scheme and a depth map using the SLscheme, and the plurality of beams may be scanned once, therebyobtaining both a depth map using the ToF scheme and a depth map usingthe SL scheme.

To detect reflections of a plurality of projected beams, such as in step320 of FIG. 3, the beam detector 120 may detect a plurality of secondbeams B2. The beam detector 120 may detect a waveform or phase of thedetected second beams B2, and the detected waveform or phase may be usedto obtain the ToF. The beam detector 120 may detect the brightness ofthe detected second beams B2. The brightness of each of the plurality ofpixels of the 2D camera constituting the beam detector 120 may be storedin the coordinates (x=f(θ), y=f(φ)) of the corresponding pixel.Accordingly, the device 100 may obtain the images corresponding to thesecond beams B2, and the device 100 may obtain the pattern reflected bythe scene or object 150 by integrating the plurality of imagescorresponding to the plurality of second beams B2.

The beam detector 120 may obtain the reflected pattern from a singleimage generated by integrating optical data related to the plurality ofsecond beams B2 obtained while the plurality of second beams B2 arereceived, rather than by integrating the plurality of imagescorresponding to the plurality of second beams B2. Integrating theoptical data related to the plurality of second beams B2 may includeexposing the 2D camera while the beam detector 120 receives theplurality of second beams B2.

Identifying the sectors corresponding to the second beams B2 among theplurality of sectors of the pattern for projecting beams, projecting thefirst beam B1 to obtain the ToF, and detecting the second beam B2 may beperformed in synchronization with each other. That is, steps 310 and 320of FIG. 3 may be performed at substantially the same time. For example,the operations of the beam projector 110 (i.e., the operations of thelaser 111 and the second mirror 113) and the operations of the beamdetector 120 may be synchronized with each other. A sync signal may beused to synchronize the operation of the beam projector 110 with theoperation of the beam detector 120. The sync signal may be generated by,or under the control of, the controller 130.

FIG. 12 illustrates a method for projecting a beam using a sync signalaccording to an embodiment.

Referring to FIG. 12, a plurality of beams 1210 may be projected inperiods in synchronization with a sync signal Sync and according to awaveform of the sync signal Sync. The period for projecting the beams1210 may vary, as illustrated in FIG. 12. The period for projecting thebeams 1210 may be based on the resolution of the pattern, and when thepattern has sectors with different resolutions, the period may vary. Forexample, the period for projecting the beams differs between a firstperiod P1 and a second period P2. The period for projecting the beams1210 in the first period P1 is shorter than the period in the secondperiod P2. Therefore, the resolution of the pattern in the first periodP1 may be higher than the resolution of the pattern in the second periodP2. The controller 130 may change the frequency of the sync signal Syncto adjust the period for projecting the beams 1210 depending on thevariation in resolution. That is, the frequency of the sync signal Syncmay increase in order to increase the period for projecting the beams1210, and the frequency of the sync signal Sync may decrease in order todecrease the period for projecting the beams 1210.

The sync signal Sync may be provided to the beam projector 110 and alsoto the beam detector 120. Thus, the beam detector 120 may identify,based on the sync signal Sync, a sector among the plurality of sectorsdividing the pattern corresponding to a first beam B1 from which thesecond beam B2 detected is originated.

A modulation control signal SL Mod. may be used to control themodulation of the beams 1210. The beam projector 110 may control themodulation applied to the beams 1210 according to the modulation controlsignal SL Mod. For example, different modulation schemes may apply tothe beams 1210, when the modulation control signal SL Mod. is low andwhen the modulation control signal SL Mod. is high. The modulationcontrol signal SL Mod. may be set to have a value of “low” correspondingto the darker sectors 1220 of the pattern and a value of “high”corresponding to the brighter sectors 1230 of the pattern, or viceversa, according to an embodiment.

In step 330, the ToFs may be identified based on the detected secondbeams B2. The detected second beams B2 used to identify the ToFs mayalso be used to detect the distortion of the pattern as set forth below.

FIG. 13 illustrates a method for detecting ToF according to anembodiment.

Referring to FIG. 13, a delay A of the second beam B2 from the firstbeam B1 may be generated by comparing a waveform W1 of the first beam B1emitted from the beam projector 110 with a waveform W2 of the secondbeam B2 detected by the beam detector 120, or by comparing a phase ofthe first beam B1 with a phase of the second beam B2. When the period ofthe first beam B1 and the second beam B2 is T, the ToF of beam may beobtained using Equation (1) below.

ToF=T×Δ/(2×π)   (1)

The distance D to where the beam is reflected may be obtained based onthe ToF, using Equation (2) below.

D=(ToF×c)/2   (2)

In Equation (2), c refers to the speed of light.

According to an embodiment, the controller 130 may obtain a variation inbrightness by differentiating the brightness of the image obtained bythe 2D camera of the beam detector 120 and may obtain the arrival timeof the beam based on the variation in brightness. The controller 130 maycalculate the delay θ from the difference between the obtained beamarrival time and the time the beam is emitted.

Accordingly, the respective ToFs of the plurality of beams may beobtained, and the respective distances (Ds) for the plurality of beamsmay be obtained based on the obtained plurality of ToFs. A ToF-baseddepth map may be generated based on the plurality of Ds.

In the step 340, the pattern distorted by the scene or object 150 may beobtained based on the plurality of second beams B2 detected by the beamdetector 120. All, or at least some, of the plurality of detected secondbeams B2, which are used to obtain the distorted pattern, may be used toidentify the ToFs in the step 330. As such, the plurality of secondbeams B2 detected may be jointly used to obtain the ToFs and the patterndistortion, thereby allowing both a ToF scheme-based depth map and an SLscheme-based depth map to be obtained in a simplified manner.

A reflected pattern (i.e., the pattern distorted by the scene or object150) may be obtained by integrating the plurality of images for thescene 150 obtained by the beam detector 120, e.g., a 2D camera. Here,each of the plurality of images may correspond to a corresponding one ofthe plurality of second beams B2.

Integrating the plurality of images may include summating or averagingthe brightness of the pixels with the same coordinates of the pluralityof images. The pattern distortion may be identified by comparing thereflected pattern obtained with the original pattern. The distance tothe scene or object 150 corresponding to each of the plurality ofsectors of the pattern may be obtained based on the pattern distortion.The less pattern distortion, the larger the distance to the scene orobject 150 may be identified. Thus, a depth map may be generated basedon pattern distortion. The pattern distortion-based depth map may alsobe referred to as an SL-based depth map.

FIG. 14 illustrates a method for obtaining a distance to an object basedon SL according to an embodiment.

Referring to FIG. 14, the beam B1 emitted from the origin Op of a beamprojector is reflected at a reflection point R of a scene or object andis detected by a beam detector. The position P corresponding to the beamB1 on the original pattern and the position C corresponding to thereflected beam B2 on the image detected by the beam detector may be inthe same epipole line. The distance D to the reflection point R may beobtained using Equation (3) below.

D=(f×B)/d   (3)

In Equation (3), f refers to the focal length of the 2D camera of thebeam detector, and B refers to the length of a baseline between theorigin Op of the beam projector and the origin Oc of the 2D camera. ‘d’denotes the disparity between the position P corresponding to the beamB1 on the original pattern and the position C corresponding to thereflected beam B2 on the image detected by the beam detector.

When the original pattern and the reflected pattern are set to have thesame resolution, d may be expressed as the disparity between the xcoordinate (Px) of the position P corresponding to the beam B1 on theoriginal pattern and the x coordinate (Cx) of the position Ccorresponding to the reflected beam B2 on the reflected pattern.

A depth map, e.g., as generated in step 350 of FIG. 3, may be generatedby merging the above-mentioned first depth map, which is created basedon ToF, and second depth map, which is created based on patterndistortion. The first depth map and the second depth map may be mergedby various methods.

For example, the first depth map and the second depth may be merged by aweighted sum. As the distance to the scene or object 150 decreases foreach pixel, a lower weight may be assigned to the distance of the firstdepth map, and a higher weight to the second depth map. As the distanceto the scene or object 150 increases for each pixel, a higher weight maybe assigned to the distance of the first depth map, and a lower weightto the second depth map. The distance to the scene or object 150 todetermine the weights may be determined by the distance of the firstdepth map, the distance of the second depth map, a mean of the distancesof the first depth map and the second depth map, or a weighted sum ofthe distances of the first depth map and the second depth map. Whengenerated by merging the ToF-based depth map and the patterndistortion-based depth map, the depth map may reduce errors regardlessof whether it is within a short or long distance.

According to an embodiment, the merged depth map may be obtained byusing Equation (4) below.

$\begin{matrix}{{z\left( {x,y} \right)} = \left\{ {\begin{matrix}{{D\; 1\left( {x,y} \right)},{{{if}\mspace{14mu} \sigma \; {D\left( {x,y} \right)}} < {\sigma \; {T\left( {x,y} \right)}}}} \\{{T\; 1\left( {x,y} \right)},{otherwise}}\end{matrix}.} \right.} & (4)\end{matrix}$

In Equation (4), z(x,y) refers to a depth at a pixel with thecoordinates (x,y) of the merged depth map. D1 (x,y) refers to a depth ata pixel with the coordinates (x,y) of the second depth map generated bythe SL scheme. T1 (x,y) refers to a depth at a pixel with thecoordinates (x,y) of the first depth map generated by the ToF scheme.σD(x,y) refers to a standard depth deviation obtained based on depths ofneighboring pixels of the pixel with the coordinates (x,y) of the seconddepth map. σT(x,y) refers to a standard depth deviation obtained basedon depths of neighboring pixels of the pixel with the coordinates (x,y)of the first depth map.

For example, the merged depth map may be generated using ultra-highresolution techniques based on Markov random fields.

Merging the first depth map and the second depth map is not limited tothe above-described methods, but may optionally adopt other variousknown methods, their variations, or newly developed methods.

When the first depth map and the second depth map differ in resolution,a process for matching the resolution of the first depth map with theresolution of the second depth map may performed prior to merging thefirst depth map and the second depth map.

The quality of the generated depth map may be identified aftergenerating the depth map based on ToF and pattern distortion.

For example, when the identified quality of the depth map generated inFIG. 3 is worse than a threshold, steps 310, 320, 330, 340, and 350 maybe performed again using a pattern different than the prior pattern, andsuch a process may be repeated until a depth map with sufficientquality, i.e., equal to or greater than the threshold, is obtained.

The pattern used for the repeated process may be completely differentfrom the prior pattern or a modification of the prior pattern may beused (e.g., one resulting from modifying the resolution of all or partof the prior pattern or one shifted from the prior pattern).

As described above, a method and a device for obtaining a 3D image,according to embodiments, may simultaneously obtain ToF and patterndistortion by point-to-point beam projection and detection, whichalleviates the calculation load of conventional processes. Further, thepoint-to-point beam projection and detection may modify or alter thepattern for projecting beams in a simplified manner.

As is apparent from the foregoing description, various embodiments ofthe disclosure present at least the following effects:

A high-quality depth map may be obtained in a wide depth range.

A method may be provided for obtaining a depth map in an adaptivelyvariable manner.

A load of calculations in generating a high-quality depth map may bereduced.

While the present disclosure has been particularly shown and describedwith reference to certain embodiments thereof, it will be understood bythose of ordinary skill in the art that various changes in form anddetails may be made therein without departing from the spirit and scopeof the present disclosure as defined by the following claims and theirequivalents.

What is claimed is:
 1. A method for obtaining a three-dimensional (3D)image by a device including a beam projector and a beam detector, themethod comprising: sequentially projecting, from the beam projector, aplurality of beams to an object, each of the plurality of projectedbeams corresponding to a respective one of a plurality of sectorsincluded in a pattern; detecting, by the beam detector, a plurality ofbeams reflected off of the object corresponding to the plurality ofprojected beams; identifying time-of-flight (ToF) of each of theplurality of projected beams based on the plurality of detected beams;identifying a distortion of the pattern, which is caused by the object,based on the plurality of detected beams; and generating a depth map forthe object based on the distortion of the pattern and the ToF of each ofthe plurality of projected beams, wherein the plurality of detectedbeams are commonly used to identify the ToF and the distortion of thepattern.
 2. The method of claim 1, wherein generating the depth mapcomprises: generating a first depth map based on the identified ToF;generating a second depth map based on the identified distortion of thepattern; and generating the depth map based on the first depth map andthe second depth map.
 3. The method of claim 1, wherein identifying thedistortion of the pattern comprises: obtaining a reflected pattern bythe object based on the plurality of detected beams; comparing thereflected pattern with the pattern; and identifying the distortion ofthe pattern based on a result of the comparing.
 4. The method of claim1, wherein the plurality of sectors include a first sector correspondingto a first resolution, and a second sector corresponding to a secondresolution, which is higher than the first resolution, wherein a firstbeam corresponding to the first sector is projected according to a firstperiod, and a second beam corresponding to the second sector isprojected according to a second period, and wherein the second period isshorter than the first period.
 5. The method of claim 4, wherein thesecond sector corresponds to a region of interest (ROI).
 6. The methodof claim 1, wherein each of the plurality of projected beams ismodulated based on a corresponding one of the plurality of sectors. 7.The method of claim 1, wherein the plurality of detected beams aredetected in synchronization with sequential projection of the pluralityof projected beams, based on a sync signal.
 8. The method of claim 7,wherein the plurality of sectors include a first sector corresponding toa first resolution and a second sector corresponding to a secondresolution, which is higher than the first resolution, wherein uponprojecting a first beam corresponding to the first sector, the syncsignal has a first frequency, and upon projecting a second beamcorresponding to the second sector, the sync signal has a secondfrequency, and wherein the first frequency is lower than the secondfrequency.
 9. The method of claim 1, further comprising generating thepattern by modifying an original pattern, wherein modifying the originalpattern comprises at least one of shifting the original pattern,modifying a resolution of all of the original pattern, or modifying aresolution of part of the original pattern.
 10. The method of claim 1,wherein the plurality of sectors include a first sector and a secondsector, and wherein a first beam projected corresponding to the firstsector and a second beam projected corresponding to the second sectorare modulated differently.
 11. A device for obtaining a 3D image, thedevice comprising: a beam projector configured to sequentially project aplurality of beams to an object, each of the plurality of projectedbeams corresponding to a respective one of a plurality of sectorsincluded in a pattern; a beam detector configured to detect a pluralityof beams reflected off of the object corresponding to the plurality ofprojected beams; and a controller configured to: identify time-of-flight(ToF) of each of the plurality of projected beams based on the pluralityof detected beams, identify a distortion of the pattern, which is causedby the object, based on the plurality of detected beams, and generate adepth map for the object based on the distortion of the pattern and theToF of each of the plurality of projected beams, wherein the pluralityof detected beams are commonly used to identify the ToF and thedistortion of the pattern.
 12. The device of claim 11, wherein thecontroller is further configured to: generate a first depth map based onthe identified ToF, generate a second depth map based on the identifieddistortion of the pattern, and generate the depth map based on the firstdepth map and the second depth map.
 13. The device of claim 11, whereinthe controller is further configured to identify the distortion of thepattern by obtaining a reflected pattern based on the plurality ofdetected beams, comparing the reflected pattern with the pattern, andidentifying the distortion of the pattern based on a result of thecomparing.
 14. The device of claim 11, wherein the plurality of sectorsinclude a first sector corresponding to a first resolution, and a sectorcorresponding to a second resolution, which is higher than the firstresolution, wherein a first beam corresponding to the first sector isprojected according to a first period, and a second beam correspondingto the second sector is projected in a second period, and wherein thesecond period is shorter than the first period.
 15. The device of claim14, wherein the second sector corresponds to a region of interest (ROI).16. The device of claim 11, wherein each of the plurality of projectedbeams is modulated based on a corresponding one of the plurality ofsectors.
 17. The device of claim 11, wherein the plurality of detectedbeams are detected in synchronization with the projection of theplurality of projected beams, based on a sync signal.
 18. The device ofclaim 17, wherein the plurality of sectors include a first sectorcorresponding to a first resolution and a second sector corresponding toa second resolution, which is higher than the first resolution, whereinupon projecting a first beam corresponding to the first sector, the syncsignal has a first frequency, and upon projecting a second beamcorresponding to the second sector, the sync signal has a secondfrequency, and wherein the first frequency is lower than the secondfrequency.
 19. The device of claim 11, wherein the controller is furtherconfigured to generate the pattern by modifying an original pattern, andwherein modifying the original pattern comprises at least one ofshifting the original pattern, modifying a resolution of all of theoriginal pattern, or modifying a resolution of part of the originalpattern.
 20. The device of claim 11, wherein the plurality of sectorsinclude a first sector and a second sector, and wherein a first beamprojected corresponding to the first sector and a second beam projectedcorresponding to the second sector are modulated differently.